A human evaluation of English-Irish statistical and neural machine translation

With official status in both Ireland and the EU, there is a need for high-quality English-Irish (EN-GA) machine translation (MT) systems which are suitable for use in a professional translation environment. While we have seen recent research on improving both statistical MT and neural MT for the EN-GA pair, the results of such systems have always been reported using automatic evaluation metrics. This paper provides the first human evaluation study of EN-GA MT using professional translators and in-domain (public administration) data for a more accurate depiction of the translation quality available via MT

Leveraging backtranslation to improve machine translation for Gaelic language

Irish and Scottish Gaelic are similar but distinct languages from the Celtic language family. Both languages are underresourced in terms of machine translation (MT), with Irish being the better resourced. In this paper, we show how backtranslation can be used to harness the resources of these similar low-resourced languages and build a Scottish-Gaelic to English MT system with little or no highquality bilingual data

A crowd-sourcing approach for translations of minority language user-generated content (UGC)

Data sparsity is a common problem for machine translation of minority and less-resourced languages. While data collection for standard, grammatical text can be challenging enough, efforts for collection of parallel user-generated content can be even more challenging. In this paper we describe an approach to collecting English↔Irish translations of user-generated content (tweets) that overcomes some of these hurdles. We show how a crowd-sourced data collection campaign, which was tailored to our target audience (the Irish language community), proved successful in gathering data for a niche domain. We also discuss the reliability of crowd-sourcing English↔Irish tweet translations in terms of quality by reporting on a self-rating approach along with qualified reviewer ratings

An investigation of English-Irish machine translation and associated resources

As an official language in both Ireland and the European Union (EU), there is a high demand for English-Irish (EN-GA) translation in public administration. The difficulty that translators currently face in meeting this demand leads to the need for reliable domain-specific user-driven EN-GA machine translation (MT). This landscape provides a timely opportunity to address some research questions surrounding MT for the EN-GA language pair. To this end, we assess the corpora available for training data-driven MT systems, including publicly-available data, data collected through EU-supported data collection efforts and web-crawling, showing that though Irish is a low-resource language it is possible to increase the corpora available through concerted data collection efforts. We investigate how increased corpora affect domain-specific (public administration) statistical MT (SMT) and neural MT (NMT) systems using automatic metrics. The effect that different SMT and NMT parameters have on these automatic values is also explored, using sentence-level metrics to identify specific areas where output differs greatly between MT systems and providing a linguistic analysis of each. With EN-GA SMT and NMT automatic evaluation scores showing inconclusive results, we investigate the usefulness of EN-GA hybrid MT through the use of monolingual data as a source of artificial data creation via backtranslation. We evaluate these results using automatic metrics and linguistic analysis. Although results indicate that the addition of artificial data did not have a positive impact on EN-GA MT, repeated experiments involving Scottish Gaelic show that the method holds promise, given suitable conditions. Finally, given that the intended use-case of EN-GA MT is in the workflow of a professional translator, we conduct an in-depth human evaluation study for EN-GA SMT and NMT, providing a human-derived assessment of EN-GA MT quality and comparison of EN-GA SMT and NMT. We include a survey of translator opinions and recommendations surrounding EN-GA SMT and NMT as well as an analysis of data gathered through the post-editing of MT output. We compare these results to those generated automatically and provide recommendations for future work on EN-GA MT, in particular with regards to its use in a professional translation workflow within public administration

SMT versus NMT: preliminary comparisons for Irish

In this paper, we provide a preliminary comparison of statistical machine translation (SMT) and neural machine translation (NMT) for English→Irish in the fixed domain of public administration. We discuss the challenges for SMT and NMT of a less-resourced language such as Irish, and show that while an out-of-the-box NMT system may not fare quite as well as our tailor-made domain-specific SMT system, the future may still be promising for EN→GA NM

Is all that glitters in MT quality estimation really gold standard?

Human-targeted metrics provide a compromise between human evaluation of machine translation, where high inter-annotator agreement is difficult to achieve, and fully automatic metrics, such as BLEU or TER, that lack the validity of human assessment. Human-targeted translation edit rate (HTER) is by far the most widely employed human-targeted metric in machine translation, commonly employed, for example, as a gold standard in evaluation of quality estimation. Original experiments justifying the design of HTER, as opposed to other possible formulations, were limited to a small sample of translations and a single language pair, however, and this motivates our re-evaluation of a range of human-targeted metrics on a substantially larger scale. Results show significantly stronger correlation with human judgment for HBLEU over HTER for two of the nine language pairs we include and no significant difference between correlations achieved by HTER and HBLEU for the remaining language pairs. Finally, we evaluate a range of quality estimation systems employing HTER and direct assessment (DA) of translation adequacy as gold labels, resulting in a divergence in system rankings, and propose employment of DA for future quality estimation evaluations

Tapadoir: developing a statistical machine translation engine and associated resources for Irish

Tapadoir (from the Irish ´ tapa ‘fast’ and the nominal suffix -oir ´ ) is a statistical machine translation (SMT) project, funded by the Irish government. This work was commissioned to help government translators meet the translation demands which have arisen from the Irish language’s status as an official EU and national language. The development of this system, which translates English into Irish (a morphologically rich, low-resourced minority language), has produced an interesting set of challenges. These challenges have inspired a creative response to the lack of data and NLP tools available for the Irish language and have also resulted in the development of new resources for the Irish linguistic and NLP community. We show that our SMT system out-performs Google TranslateTM (a widely used general-domain SMT system) as a result of steps we have taken to tailor translation output to the user’s specific needs

A human evaluation of English-Irish statistical and neural machine translation

With official status in both Ireland and the EU, there is a need for high-quality English-Irish (EN-GA) machine transla- tion (MT) systems which are suitable for use in a professional translation environ- ment. While we have seen recent research on improving both statistical MT and neu- ral MT for the EN-GA pair, the results of such systems have always been reported using automatic evaluation metrics. This paper provides the first human evaluation study of EN-GA MT using professional translators and in-domain (public adminis- tration) data for a more accurate depiction of the translation quality available via MT

Procedural and physical interventions for vaccine injections systematic review of randomized controlled trials and quasi-randomized controlled trials

Background: This systematic review evaluated the effectiveness of physical and procedural interventions for reducing pain and related outcomes during vaccination. Design/Methods: Databases were searched using a broad search strategy to identify relevant randomized and quasi-randomized controlled trials. Data were extracted according to procedure phase (preprocedure, acute, recovery, and combinations of these) and pooled using established methods. Results: A total of 31 studies were included. Acute infant distress was diminished during intramuscular injection without aspiration (n=313): standardized mean difference (SMD) -0.82 (95% confidence interval [CI]: -1.18, -0.46). Injecting the most painful vaccine last during vaccinations reduced acute infant distress (n=196): SMD -0.69 (95%CI: -0.98, -0.4). Simultaneous injections reduced acute infant distress compared with sequential injections (n=172): SMD -0.56 (95%CI: -0.87, -0.25). There was no benefit of simultaneous injections in children. Less infant distress during the acute and recovery phases combined occurred with vastus lateralis (vs. deltoid) injections (n=185): SMD -0.70 (95%CI: -1.00, -0.41). Skin-to-skin contact in neonates (n=736) reduced acute distress: SMD -0.65 (95% CI: -1.05, -0.25). Holding infants reduced acute distress after removal of the data from 1 methodologically diverse study (n=107): SMD -1.25 (95% CI: -2.05, -0.46). Holding after vaccination (n=417) reduced infant distress during the acute and recovery phases combined: SMD -0.65 (95% CI: -1.08, -0.22). Self-reported fear was reduced for children positioned upright (n=107): SMD -0.39 (95% CI: -0.77, -0.01). Non-nutritive sucking (n=186) reduced acute distress in infants: SMD -1.88 (95% CI: -2.57, -1.18). Manual tactile stimulation did not reduce pain across the lifespan. An external vibrating device and cold reduced pain in children (n=145): SMD -1.23 (95% CI: -1.58, -0.87). There was no benefit of warming the vaccine in adults. Muscle tension was beneficial in selected indices of fainting in adolescents and adults. Conclusions: Interventions with evidence of benefit in select populations include: no aspiration, injecting most painful vaccine last, simultaneous injections, vastus lateralis injection, positioning interventions, non-nutritive sucking, external vibrating device with cold, and muscle tension

A National Spinal Muscular Atrophy Registry for Real-World Evidence.

BACKGROUND: Spinal muscular atrophy (SMA) is a devastating rare disease that affects individuals regardless of ethnicity, gender, and age. The first-approved disease-modifying therapy for SMA, nusinursen, was approved by Health Canada, as well as by American and European regulatory agencies following positive clinical trial outcomes. The trials were conducted in a narrow pediatric population defined by age, severity, and genotype. Broad approval of therapy necessitates close follow-up of potential rare adverse events and effectiveness in the larger real-world population. METHODS: The Canadian Neuromuscular Disease Registry (CNDR) undertook an iterative multi-stakeholder process to expand the existing SMA dataset to capture items relevant to patient outcomes in a post-marketing environment. The CNDR SMA expanded registry is a longitudinal, prospective, observational study of patients with SMA in Canada designed to evaluate the safety and effectiveness of novel therapies and provide practical information unattainable in trials. RESULTS: The consensus expanded dataset includes items that address therapy effectiveness and safety and is collected in a multicenter, prospective, observational study, including SMA patients regardless of therapeutic status. The expanded dataset is aligned with global datasets to facilitate collaboration. Additionally, consensus dataset development aimed to standardize appropriate outcome measures across the network and broader Canadian community. Prospective outcome studies, data use, and analyses are independent of the funding partner. CONCLUSION: Prospective outcome data collected will provide results on safety and effectiveness in a post-therapy approval era. These data are essential to inform improvements in care and access to therapy for all SMA patients